Inter-frame flickering is a commonly seen artifact in encoded video signals, especially at periodic I-frames in low or medium bit rate coding, which greatly degrades the overall perceptual quality of the coded video. How to accurately measure the flickering artifact is an important issue in practice. Existing flicker metrics are typically rough solutions suitable only for measuring I-frame flickering of the whole sequence to provide a single measure of I-frame flicker that is descriptive of the whole sequence, and fail when measuring a per-frame flickering artifact (that is, they fail to provide a single measure for a given frame).
Inter-frame flickering is a commonly seen artifact in video coding, as well. Although they are visually identical in the original video signal, the same static background areas in two consecutive video frames may look different in the encoded/reconstructed video. This is because the areas are encoded through different encoding processes with different coding parameters, for example, different macro-block (MB), quantization parameters (QP), and/or different MB coding modes (such as the Inter or Intra mode). Consequently, flickering will be perceived when playing out the coded video.
Flickering is more easily perceived in static or low motion areas of video frames than in medium or high motion areas, because of the well-known motion masking effect of human visual systems (HVS). In addition, flickering is more prominent in low or medium bit rate coding and/or at periodic I-frames in a coded sequence. While the previous frame is coded as a P-frame using inter-frame prediction, the current frame is coded as an I-frame with all the MBs, using intra-frame prediction. Coarse quantization is often used due to limited bit rate. Accordingly, a large reconstruction difference may result for the corresponding static MBs collated in the two frames, and hence, a serious I-frame flickering or pulsing artifact may result. Flickering artifacts, especially I-frame flickering, greatly compromise the perceptual quality of coded video.
How to accurately measure the flickering artifact is an important issue in practice. A good method for measuring flickering would not only prove valuable to evaluate the performance of various de-flickering schemes, but also, perhaps more importantly, such a method can be exploited to develop various encoder optimization techniques to more effectively reduce flickering. For example, rate-distortion (RD) optimization is a commonly adopted framework in practice to make encoding decisions over a variety of coding options, for example, macro-block (MB), motion vector, and coding mode, etc. Conventional RD optimization only considers the distortion between the reconstructed video and the original video. To reduce flickering, and hence, improve perceptual video coding quality, one can further introduce the flickering distortion in RD optimizations.
A widely used method to measure flicker is the so-called S metric, which measures the overall I-frame flickering artifact of an entire coded video sequence. It is defined as follows.
                              D                      Flicker            ,            l                          =                              1                          M              ·                              N                static_MBs                                              ⁢                                    ∑                              1                -                                  frames                  :                  n                                                      ⁢                                                  ⁢                                          ∑                                  static                  ⁢                                                                          ⁢                                      MBs                    :                    i                                                              ⁢                                                ∑                                      pixels                    :                    j                                                  ⁢                                                                                                                      f                        ^                                                                    n                        ,                        i                                            j                                        -                                                                  f                        ^                                                                                              n                          -                          1                                                ,                        i                                            j                                        -                                          (                                                                        f                                                      n                            ,                            i                                                    j                                                -                                                  f                                                                                    n                              -                              1                                                        ,                            i                                                    j                                                                    )                                                                                                                                                (        1        )            
Herein, the concerned frames involve all the non-scene-change I-frames in a sequence. That is, frame n denotes a non-scene-change I-frame and frame n−1 is the frame played out before frame n. Note that scene-change I-frames are excluded, as flickering typically only occurs for frames in the same scene. Additionally, fn,ij and {circumflex over (f)}n,ij denote the original and the reconstructed pixel j of MB i in frame n, respectively. M denotes the total number of pixels in a MB, and Nstatic—MBs denotes the total number of static MBs in all the non-scene-change I-frames. Flickering is assumed to occur only on static MBs, which is defined as follows.
MB i in frame n is a static MB, if its mean-absolute-difference (MAD) is small, that is:
                              MAD                      n            ,            i                          =                                            1              M                        ⁢                                          ∑                                  pixels                  :                  j                                            ⁢                                                          ⁢                                                                                    f                                          n                      ,                      i                                        j                                    -                                      f                                                                  n                        -                        1                                            ,                      i                                        j                                                                                                <                      ɛ            .                                              (        2        )            
In practice, parameter ε could be, for example, 10. Note that instead of MAD, (1) and (2) can also be defined based on another measure, such as, for example, mean-square-difference (MSD) with parameter ε set to, for example, 500.
The S metric has been widely used to evaluate the performance of various I-frame de-flickering schemes. However, in practice, via experiments, the S metric is actually a fairly rough model. For example, it is only applicable to measure the overall I-frame flickering of an entire sequence, and fails when applied to generally measure flickering of each single frame. Note that the S metric is intended to be applied to a sequence, as evidenced by the summation in Equation 1 over a set of I frames. Removing that summation results in a metric that provides a result on a per-frame basis, but that result is not reliable as discussed below.
One fundamental defect of the metric is that using |{circumflex over (f)}n,ij−{circumflex over (f)}n-1,ij−(fn,ij−fn-1,ij)| to measure flicker at each pixel is only accurate if |fn,ij−fn-1,ij| is small (for example, less than 10). However, even with a small MAD for the entire MB, |fn,ij−fn-1,ij| of some particular pixels could be large (for example, around 100). In this case, small |{circumflex over (f)}n,ij−{circumflex over (f)}n-1,ij| pixels which actually render little flickering, will lead to large |{circumflex over (f)}n,ij−{circumflex over (f)}n-1,ij−(fn,ij−fn-1,ij)|, that is, large flickering distortion in the metric. This contradicts the actual viewing experience, and compromises the accuracy of the flickering model. As an example, FIG. 1 shows the result of applying Equation (1) to measure the per-frame flickering artifact (by removing the summation over a set of I-frames). FIG. 1 shows flickering distortion along the vertical axis 20 versus frame number along the horizontal axis 10 using an existing flickering metric. The sequence is the 150 frames CIF sequence “Bus” with 30 f/s, and coded at 300 kb/s with 30 frames per group-of-picture (GOP). All the non-scene-change I-frames are marked out with circles 30, and the rest of the frames are all P-frames. Visually, significant flickering is observed at the indicated I-frames, while little flickering at the P-frames. However, as seen in FIG. 1, the existing metric fails to truthfully reflect that difference.
Another drawback of the existing metric is that the static MBs are MBs at exactly the same location in the two consecutive frames. That is, no motion compensation is involved. However, a flickering artifact may be perceivable not only at completely static areas, but also at low motion areas. Hence, low motion MBs should also be included in the flickering distortion calculation. However, without motion compensation, even very low motion MBs may have a large inter-frame MB difference. Thus, such low motion MBs are inappropriately excluded from the calculation, which also compromises the accuracy of the metric.